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AESTRACT 

This paper reports on aa experiment desiqned to 
investigate the effect of structural and linguistic var-Ldbles on 
level of difficulty in solving arithmetic word problems. 
Identification of such variabl3S is intended to assist curriculum 
writers in preparing exercises at a specified level of difficulty tor 
students at various age levels. The s^.udy also considers the 
variables under varying time conditions and 6eeks to devise a coding 
system of the linguistic variables that would improve the accuracy of 
a linear regression^model previously used in *^iinilar investigations. 
Details of the theory and methodology of the experiment are provided, 
and the results are discussed. The signif i ::ance of thf, structural and 
lincuistic variables is noted. (VM) 
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Abstract 

Structural and Linguistic Variables 
in Problem Solving 

by 

Hax Jerman 
Sanford M i rman 
The Pennsylvania State University 



C OVC ATtOM * mML^MWU 
tOUCATlON 

A» Sfj |»OtNTSf« y If ^ OB OP'NK.'NS 
.''*'. t- rX) NO* Nf r F SS*.Hik V P«t 



Three sections of a methods course for prospective elementary school 

teachers were ^iven a set of word problem exercises under different time 

I* 

condit'ons; 20 minutes; one hour; and one day. Structural variables, which 
accounted for a significant amount of the observed' var iance in the error 
rate in arithmetic word problems in Jerman's (1972) study using students 
in Grades ^-9, -v^ere not consistent in entry in a stepwise regression nor 
did these variables account for a significant amount of the total variance. 
Linguistic variables* however^ which were used in Krushinski's (1973) study, 
showed consistency in ear I^y entry and accorunted for a reasonable amount of 
the variance i n. Jthc^observed proportion correct. The time effect appeared, 
however, to influence order of entry more for structural than linguistic 



variables. 



Structural and Linguistic Variables 
in Problem Solving 

by 

Max Jerman 
Sanford Mirman 
The Pennsylvania State University 

Previous studies have attempted to identify and define structural and 

linguistic variables in word problem exercises which account for a large 

amount of the observed variance in the proportion correct {Suppes, Loftus, 

andjJerman, 1969; Loftus. 1970; Jerman and Rees, 1972; Jerman, I972'a; 

Jerman, 1972; Krushinski, I973)* 

One of the goals of these studies is to identify and quantify, a small 

set of independent structural variables which will permit curriculum wri ters 

« 

to prepare exercises at a specified level of difficulty for students at 
various age Jevels. A primary task, in these studies is, therefore, to develop 
a robust set of variables. 

Y 

One purpose of this study was to determine if a Sict of structural 
variables which was found to account for a significant amount of the 

y 

variance in the observea proport ion correct for word-problem exercises 
solved by students in grades ^-9 (Jerm^n, 1972) would also account for 
a significant amount of the observed variance for a different set of word • 
problems solved by col lege- level students. 

A second purpose of th?s study was to determine whether or not a 
modification of the coding of the linguistic variables used in the Krushinski 
(1973) study would inj^ove th^ accuracy of the linear regression model's 
prediction. ,» 

A third purpose of the study was to test the robustness of the 
variables under varying tiirre conditions. 
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The six structural variables given in Je^rman (1972) and used in this 
study are defined as foHows: 

Convers ion ; A count of one v/as assigned if a conversion of units 
was required to solve the problem, zero otherwise. 

Example - If It wa:5 necessary to change hours to seconds 
to solve the problem, Conversion =1, 

2. Reca I ; The su)n of the f ol 1 owj r g: 

a. . One count for a formula to ^e recalled arid a count for each 

step in the formula. 

b. One count for each conversion to be recalled and used. 

c. One count for each fact from a previous problem to be recalled 
and used . . 

Er^mple - If the formula A « 2(l+w) was required but 
no conversions or facts from previous problems were 
. needed , 

Recal =» I (area formula) + 2 (addition and multiplication steps) + 0 
(conversions) +0 (facts) = 3. . 

3. -Length : The number of words in t>ie problem*' 

4. Operations 3 (0PER3): The sum of the following: ' ' ^ 

a. The number of different operations^ 

b. Four times the number of occurrences of the operation of 
division. 

c. Two times the number of occurences of the operation of 
multipl icat ion." 

d. The number of occurrences of the operation of addition. 
Example If the solution wa: of the form. 

■( [I6x30)xi<0] 7 3 + 1),. 



3 - 

./ 

0PER3 ■ 3(3 different operations) -^Ixk (I occurrence of division) 
+ 2x2 (2 occurrences of mui t i pi i cat ion) I (I occurrenc/ of 
addition) « 12 

5. H0HC2 : A count of I was assigned for each time a regrouping 

2 

occurred in each multiplication exer<Si%€ in the problem . 

Example - 38 

23 N0MC2 « 3 S 

Tgo ' 

Regroupings - I. 5x8 ■ ^0 units = 0 unirs k tens 

2. 5x3 tep* k tens » 19 tens « I hundred t 9 ter/s / 

3. 2 t<;ns X 8 = 16 tens « I hundred 6 tens 

6* Qu vt ient (QUO): A count of I was given for each digit in the quotient 

if division was required and zero otherwise"^. 

303 

Example - 22/ 6666 • QUO - 3 
The 1 linguistic variables^ given by Krushinski (1973) and derived 
from Golub (1971), used in the study are defined i% follows: 

1. Number of words (NWD): The number of words in the problem . 

2. Number of Sentences (NSEN) ; The number of senteoces in the proolem. 
,3. Sentence Vcngth (SENLTH) : The ratio of NWD to NSEN. 



A whole number wa> given a count of I. 
Regrouping in summ\ng to obtain the product was not counted. 



the quotient did not terminate after 2 decimaJ places, a count of 3 was 

givefi for the df»cimal places (2 places and round off), 

!>^2 

Examples - 77 9.9^ QUO 3 

^ 7/Tor QUO «» ^ 

A numeral was given a count equal to the number of words in the expression 

of the numeral in words. Example - 122b - one thousand two huK^red twenty-six 
(count - 5) . 



k 

^, • Number of Main Clauses {U The number of main clauses in the 

problem. 

5. Number of Subord i nate C I au - s (NSC): The number of subordinate 
c I agses i n the prob 1 em. ^ 

6. Number of C laubes (Ud) : The sum of NMC and. NSC. 

7. Number of Words t n the Ma i n C lauses (WOMC ) : 

The number of ;•.< ' ds in the main clauses (identified by NMC) in 
the problem, 

8. Number of Woras in the Subordinate Clauses (WQSC): The number of 
words in the subordinate clauses . . * i f i ed by NSC) in the problem. 

9. Clause Length (CLTH): The ratio of NWO to NC. 

10. Main Clause length (mCLTH) The ratio c* ^ C to NMC. 

M. Subordinate Clau se Length (SCLTH); The ratio of WDSC to NSC. 

12. Number of Prooos i t iona 1 Phrases (NPP): The number of pr.opos i t i ona 1 

phrases in the problem, 
t 1 

13. Number of Words in the Question Sentence (NWQS): The number of 
words in. the question sentence. 

I A. Nupjerals in .the Question Sentenc e (NUMQS) : Assigned a value of one 
if tRe question sentence contained a numeral and a value of zero 
otherwise.* 
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Methods 



Subjects 



Seventy-six prospective el^ementary school teachers enrolled at 
Pennsylvania State University in three sections of Elementary Education 326, 
Teaching Elementary Schoolf^r ^thmet ic, participated, in the study, Mpst 
had taken on4y algebra and geometr^y'^n high school and a prerequisite course, 
Mathematics 200-Nu(Tnber Systems, at Pennsylvania State University. Sections 
I J I, and III had 29', 27, and 20. stude^rt's' enroll ed, respect i vel y. Students 
registered for the course according to how \t would fit into their scKedule. 
Scheduling was done by a co<H|juter. 

Problem Set * ' 

The Mathematics Aptitude Test (MAT-R2), published^by the Educational 

Testing Service^ was the problem set used in the study. This set consists 

of word problems requiring arithmetic or very simple algebraic concepts 

* » 

only. The test, which contains two parts, each consisting of 15 items 

; X . 

witb a time limit or 10 minutes per P^art, is suitable for students in 
grades 11-16. . ' 

• ' r 

- ' The probtws without answer choices were mimeographed, five to a page, 
with work space provided for each problem. Each lab instructor administered 

; 

the problem set at a time convenient to the schedule of his class. The^ 
test was administered to sections I and 111 in the middle af April, 1972, and 
to Section II at the beginning of May, 1972* Students in Section I had 
20 minutes during a lab session to complete the exercises, students in 
Section I I had 1 hour during a lab period to complete the problems, and 
students in Section III were allowed one day to de the problems at home. 
Students in Section III were asked not to collaborate or seek help* in 
doing the exercises, j 



The authors scored each problem for each student as either correct 

■ ) Am^ 
{correct answer) / incorrect (an attempt oi some kind was made but tliere 

was not a correct answer given) or omit (no attempt was made in solving the 

problem).. Problems which were omtt^ed were not included ir> the analysis. 

The MAT-R2 ^est was coded first for the above set of six structural 

variables and then for the above set of linguistic variables, 

A stepwise linear regression program, BMD02R (UCLA), which was 

modified to include a log transform-^and an ant i log transform to produce 

probabilities between 0 and 1, was applied to the MAT-R2 test. The 

regression program was applied for the set of six s'truc6ural variables ^ 

alone, the set of 1^ linguistic variables alone, and the set of 20 

cortbined variables to each of the three sections and to the three sectio^ns/ 

combined into a single group, in addition, an i tern analys is program, 

ITEMPGMI (Stanford), and an analysis of variance program, ANOVR (Pennsylvania 

State Unj vers i ty) .were applied to the MAT-R2 test. 

Results 

The mean total score on the test was* 13,00, with a standard, deviat ion 
of k.SkS^ Cronbach's alpha was ,82'*, with error of measurement 2,07^. The 
range of scores was 2^, from a low of zer^c^orrect to a high of 2^ cprrect, * 

The means and standard devr^tl^s for each of the three sections, 
using for each student both the\o^served proport>ion correct of the 30 ^ 
problems, disregarding omits, ^nd the number of correct responses, are given 
In Table 1, 



Insert Table 1 About Here 
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A summaryof the one-way analysis of variance procedui^, using ')Oth 
the observed proport ion xorrect and ifie number of correct responses is 
given Ln Table 



Insert Table 2 About Here 



The calculated Fof ^♦.20'*, obt<3:ped by using the observed, proportion 
cbrrectj was significant at the ,05 level. Using the number of correct 
responses produced a calculated F of 20.2^3 wWch was sighif leant at the 
,001 level. Hence, the sample data do not support, in either application 
of the- analysis of variance, the hypoth^s i s that the thx^e population means 
were equal, tt seems clear, from the data in Table that there was a^ 
di^Njtt relationship between time allowed students to solve the problems 
and the number of correct solutions per student. The mean of the number 
of correct responses for students in section III, the one day section, was 
almost 80 percent greater than the mean of the number of correct responses 
for students in section I, the 20 minute section. The number of correct / 
responses gives a better indication of the time effect on d i f f rcul ty- 1 eve I 
than docs the observed proportion correct since the latter disregards 

;fficul^y if 

i 

number of people omFt the problem. 

The order of entry of the variables, R, and for each of the first 

six steps in the* regression for each of> the three sections and for the 

three sections , combined are given in Tattles 3,^, and 5 for the six 

structural^ 1^ I inguistic, and 20 combined variables, respectively. In 
2 

addition, R a*)d R ^^or the last 'Step are also given. 



omits, thus not giving a cleqr Indication of problem difficult if a laj^^e 

0 I 



^ f 

^ Insert Tables 3,'«, and 5 About Here 
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As shown in Table 3, for the three sections combined, » .6^, 

2 ( 
with « .^1. Thcjs these six st^uitural variables accounted for only 

percent of the variance in the observed proportion of correct responds » 

The order of entry of. the variables in section I, the section with a 

26 midute time limit, was decidedly different from that of sections II and 

III. However, the ordersof entry of^ the first five variables that entered 

> 

the regr^ession for sect ions II and Ml were identical except that 'f1ie second 
and third variables were interchanged. 

The regression coefficients, standard ^^rors of regression coefficients,^ 
and computed t-values for each of the three sect ions and for the ^lK'ee\^ 
sections combin.ed are presented in Tables 6,7, and 8 For the, six structural, 
]k linguistjc, and 20 combined variables, respectively. 



Insert Tables 6-, 7,, ^ndS About Here 

r 

As indicated in Table 6, the st rue tura 1 • var i ab 1 es , which entered the 
regression, were almost never s^ignificant for a,py group tested. 

The first six variables which entered the linguistic regression analvsi* 
fcfr the three sections combined, as shown in Table jA, were NC, NUnQS,*^ * 
Cl/TH, NPP, HSEN, and WDSC.^ The . mu.) t i pi e R a.t step six was ..79, with. ^ 

■ .63i indicating that 63 per cent of the vari^QC^ in the ob>^rved 
proportion correct was accounted for by these six varlab^Ufs. It is interesting 
that three varl^ables NSEN, NUMQS, .and NC entered ip^the first six steps ^ 
in each the three sections. Especially noteworthy is'that in addition 
to these three variables, CLTH' and NPP entered in the first sIa steps in ^ 
both sections I and Ilk Henci, the five variables, NSEM, NUMQS, NC » CLTH, ' 
and NPP entered in the first Vlfi steps in both section I, the 20 minute 



section and section III, the one day section. ' ^ 

As indicated in Table 7, the first four var i ab I es, wh i ch entered the 

regression for' the three sections comb i red were significant. Although the 

six variables ^ich-^nTeVed t .regress ion in 5ecti<^n I anJ the first 

four which entered in section III were significant, none of- the first six 

variables were significant for section II, the one hour section. 

Especially significant is that the multiple P at step six for section 

II!, the one day section, was R = .86» with » .7^.^ The multiple R a^ 

> 2 • 

the last step was R^^ « .96 with := .9I, • ^ ■ 

In the 20 variable combined analysis, as show.i in Table 5» two of the 
first five variables which entered the regression for the tnree sections 
combi i^ed^ere 3tructura! variables. 0PER3, -he operatio<is variable, and 
QUO, the division variable. The rem& i n i ng f oun var i abN^es which entered 
in the first six steps were fhe linguistic variables NC , NUMQS, NPP^ 



and MCLTH. The 



2 

regress ion* produced =» .83, R^ « •67»and R^^ » .96, 



2 

with-R^ « .95. ,0f these six variables, as is shcKvn in Table 8, the Sfgnifi 
cant variables were three linguistic variables, fjCLTH, NPP, and NC, and the 
structural variable 0PER3. CLTH and NC entered in the first six steps for 
each of the three, sect ions , with UC enter i n^ »f i rst in both sec •lions II and 
111. These twi variables were significant each time they occtirred except 
that CLTH was not a s i gn i f i cant " var*i ab le for section II. Zero, two, and 
one structural variables entered the stepw*ise regression fgr sectional, il 
and Ml, respectively. 

Interestingly, of the first six variables to en\er the regrrtsion 
analysis, four J i ngui st i c 'var iaol es , NUMQS, NC, CLTH, and UPP entered in 
both section I, the 20 minute section and section I II , the one day section* 
In addition, the regression analysis produced for section Ilia multiple R 
of .86 at step 6, with - ,75. * . 



10 



Pi scuss ion 

There was a substantial time effect with respect to the ^niean number 
of correct responses for the three sections. This effect, howeve-, seemed 
to be different for the structural and linguistic variables. 
^ There was almost no corsisccncy in the order of entry of th^ six 

structural variable^ in section I, the 20-minute section, ard sectiOTi !!!, 
the One dcjy section. In toct, GPER 3, which entered first m the regression 
for section I, entered last for section 111 and QUO which entered last 
in section I entered second in section ML 

Linguistic variables^ however, were cons i stent with respect to early 
^ entry In the regression analysis for sections 1 and 111, Of the' fourteen 
linguistic variables used, five variables, UUMQS, NPP, CLTH, NC , and.NSEN 
^ entered in the first six steps for, boCti sections- In addition, these 

five variables elfrered in the first five steps in the Hnguistic regression 
for the three sections c6mbined.^ Four of these five variables, nSmQS, NPP, 
CLTH, and fiC were signifir:ant variables for, sect ion I, section III, 
^' and the three'lect ions" combi ned. No structural variables, however, were 
significant variables for any of these groups iTi the structural regression 
analysis. Three linguistic variables, NSEN, NC^land NUHQS entered in th^ 
first six steps fcr each of the three sections. Very possibly ttie time 
effect is not as important ^i'th linguistic variables as it is with structural 
variables. 

. . Therif^as a significant difference in the amount of, variance accounted 
for at step six in the structural and linguistic analysis. Disappoint ifigly» * 



the Vegrcss ion, for the three sections combined, using the six structural 

2 • ' 

variables, produced « .^1 , thus accounting for, at step 6|, only ^1 percent 

of .the total .variance in the observed proportion correct. The maltimum value 
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2 • 
obtained for in any structural regression was .^7, which occurred for 

section ill. Hence, even with a full day time limit, the six structural 

variables alone accounted for only k^ percent of the v^ariance in the observed 

'propprtion correct. - <' 

In the study by Jerman ()972)l of students in grades the first.** 

three variables which entered the stepwise regression, 0PER3, Length, 

and N0MC2, produced = .^7, thus, account i ng for 47 percent of the total 

variance in just three steps. Poss ibly,^ structural v^^iables are not as 

good predictors of difficulty lev^^^^r/^r i thmet ic word problems.^r pro- . - 

spective teachers as for elementary or junior h i gh 'schopj childreri^* 

The linguistic variables acj^punted for a greater amount of the total 

variance than did the structural variables. The nultiple R at step six for 

section 111, the one day section, for which the time effect was insignificant, 

o/ 2 - ' ' 

was •86 with fj^ « .75. This is a reasonably high percent of variance 

^2 ■ 

accounted for at step-six and i^much larger than the R produced at step six 



in any fj^egression analysis using the six structural variables alone. 
Perhaps linguistic variables are better pred ictor^ , than structural variables 
in arithmetic word problems for the population in question. 

An analysis of the 20 combined variables also indicat^LS-t^e possible 
importance of the linguistic variables. Although two structural variables, 
0PER3 and QUO, entered the regression in the first six steps for^Yhe-^hrec 
sections combined* on'v 0PER3 was significant whereas three of the 
linguistic variables which entered in the first six steps, MCLTH, NPP, and 
NC were significant. Only two structtjral variables, N0MC2, which was 
s/gnificant the one time it, occurred, and QUO, which was not significant, 
entered in the first six steps for any of the'three sections in the combined 
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20 variable analysis. In fact, only zero, two, and one structural variables 
entered the regression in the first six steps for sections I, II, and Ml, 
respect Tvel y. Two linguistic variables, however, NC and CLTH, entered 
rn the firs^C six steps for each of the three sections. In addition, 
although, there was no structural variable which entered the regression 
analysis, for the 20 combined variables, in the first six steps for both 
sections I and 111, f^^ur linguistic variables, NUMQS, NC , CLTH, and NPP, 
did. 

Perhaps the impprtance of the linguistic variables rr^ay best be seen 
by o'^servation of the results of the 20 combined variables for- sect ion III, 
the one day section. QUO was the only structural variable among the first 
six variables which entered the regression. Althou-gh QUO, was '\ot significant:, 

V 

the f i ve 1 i ngui St ic variables which entered in the first six steps were all 

» 

significant at the .005 level, except NWOQS, which was significant at 
the .025 level, • 

This study should be looked upon only as investigatory in nature. 
Future studies must be better controlled with respect to uniformity in 
ddmi n i St rat-M^n^ii^^rob 1 em sets^ a greater ratio of problems to variables 
should be included if at all practical, the order of problems should be 
randomly assigned to students in order to minimize the interaction effect 
of problem order and difficulty level, and possibly', if enough time is given 
so that students have ample opportunity to try each problem, the observed 
proportion correct could be redefined to be the ratio of correct responses 
to total number of students so that omitting a problem would reflect not 
being able to do it and not simply the lack of sufficient time to attempt 
the problem. 
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The results of this study are encouraging but not satisfactory. The 
percent of variance accounted for at step six, in any regression, was not 
sufficiently large. in addition, there obviously is a great deal of depen- 
dency between the va r i ab 1 es , espec i a 1 1 y the linguistic variables.. Hence, 
a refinement and further development of tti'ese variables is needed. A 
basic objective for future study is to produce a set of five or six 
independent variables which account for a significant amount of the 
variance in the difficulty level of the arithmetic word-problems and 
which possess the follovjing characteristics: 

1. Each variable permits unambiguous, unique, quantitative coding. 

2. Each \Jariab\e permits easy coding; i.e\ it will not be difficult 
for ""jn -^Cra i ned person to apply the definition of the vsr{ab]e 
to 



the coding of problems. a. 



3. each variable is comprehensive in its applicability to elementary 

arithmetic word -^rSb 1 ems . . 
k. There is a d i rect, a 1 though not necessarily linear, relationship 
between the njmerical value of the variable and the difficulty 
level of the problem. 
Given that such a set of variables could be coVistructed, then problems 
with a specified difficulty level could be written so that curriculum 
developers and teachers would be in a much better position to control the 
difficulty level of arithmetic word -probl ems when prepar ing instructional 
materials. \ ' 
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TABLE I 



Means and Standard Deviations for Each Section 
(hing Observed Proportion Correct 
and Number of Correct Responses 
for Each Student 
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